Multiple orientation detection

ABSTRACT

Methods, systems, computer-readable media, and apparatuses for image processing and utilization are presented. In some embodiments, an image containing at a face of a user may be obtained using a mobile device. An orientation of the face of the user within the image may be determined using the mobile device. The orientation of the face of the user may be determined using multiple stages: (a) a rotation stage for controlling a rotation applied to a portion of the image, to generate a portion of rotated image, and (b) an orientation stage for controlling an orientation applied to orientation-specific feature detection performed on the portion of rotated image. The determined orientation of the face of the user may be utilized as a control input to modify a display rotation of the mobile device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/361,405, filed Jul. 12, 2016, entitled “MULTIPLE ORIENTATIONDETECTION” which is incorporated herein by reference.

BACKGROUND

Aspects of the disclosure relate to the detection and utilization ofobject orientation associated with images captured by mobile devices.According to embodiments of the present disclosure, the detectedorientation can be utilized in various ways. One example is theautomatic control of the rotation of a display presented to a user by amobile device. Assessing appropriate display rotation can be achallenging task. Current techniques are lacking in many ways in termsof both accuracy and computational efficiency. For instance, a simpleauto display rotation feature commonly implemented in mobile devicesutilizes the direction of gravity, as measured using an accelerometer,to control the rotation of the display presented to the user (e.g.,“portrait” v. “landscape” rotation). However, proper display rotationsometimes cannot be accurately determined by simply using the directionof gravity. For example, if the mobile device is laid flat on a tablewith the display facing up toward the sky, the gravity vector is pointedin a direction perpendicular to the plane of the display and thereforeprovides no useful indication of the proper rotation of the display topresent to the user. Oftentimes, the last orientation of the displaybefore the device is laid down flat is kept in used, which may not bethe appropriate rotation for proper viewing by the user. As anotherexample, if the user holds a mobile device in front of his face whilelying down on his side (e.g., on a bed), a gravity vector-controlledauto display rotation technique would generally result in incorrectdisplay rotation from the perspective of the user. Here, the display mayautomatically rotate, incorrectly, with even small movements. This isbecause the system, using just the gravity vector, would only sense thatthe mobile device has moved from an upright position to a prone positionand conclude that the display must be rotated (e.g., from “portrait” to“landscape” rotation) in response. However, the system would not sensethat the user has also moved from an upright to a prone position, andtherefore no rotation of the display would actually be needed. These andother shortcomings of existing systems highlight the need for improvedtechniques for image-related orientation detection and utilization.

BRIEF SUMMARY

Certain embodiments are described for improved display rotation based onimage processing. A method may comprise obtaining, using a mobiledevice, an image containing a face of a user of the mobile device. Themethod may further comprising determining, using the mobile device, anorientation of the face of the user within the image. The orientation ofthe face of the user within the image may be determined using multiplestages, which may comprise: (a) a rotation stage for controlling arotation applied to a portion of the image, to generate a portion ofrotated image, and (b) an orientation stage for controlling anorientation applied to orientation-specific feature detection performedon the portion of rotated image. The method may further compriseutilizing the determined orientation of the face of the user as acontrol input to modify a display rotation of the mobile device.

In one embodiment, the portion of rotated image is stored in a rotationbuffer. In one embodiment, the orientation-specific feature detection isperformed using a computer vision (CV) computation unit. In oneembodiment, the orientation of the face of the user within the image isdetermined based on multiple modes. The multiple modes may comprise: (a)a detection mode for detecting an initial orientation of the face of theuser within an initial image and (b) a tracking mode for tracking theorientation of the face of the user within a subsequent image, using thedetected initial orientation.

According to one embodiment, in the detection mode, the initialorientation of the face of the user within the image is detected byperforming feature detection at a first plurality of hypothesizedangles. In the same embodiment, in the tracking mode, the orientation ofthe face of the user within the subsequent image is tracked byperforming feature detection at a second plurality of hypothesizedangles, the second plurality being fewer than the first plurality.

According to another embodiment, in the tracking mode, the orientationof the face of the user within the subsequent image is tracked upondetection of a trigger condition associated with on a non-image sensor.For example, the non-image sensor may comprise an accelerometer

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the disclosure are illustrated by way of example. In theaccompanying figures, like reference numbers indicate similar elements.The figures are briefly described below:

FIG. 1A depicts a scenario in which a mobile device is laid flat on topof a table and viewed by a user;

FIG. 1B is a different view of the scenario depicted in FIG. 1A, showingthe orientation of the display of the mobile device and the position ofthe user;

FIG. 2A depicts a subsequent scenario in which the mobile device remainsin the same position on the table, but the user has moved to view themobile device from a different user position, i.e., from a differentside of the table;

FIG. 2B is a different view of the scenario depicted in FIG. 2A, showingthe orientation of the display of the mobile device, in response to thechanged position of the user;

FIG. 3A depicts a potentially frustrating scenario in which a user viewsan incorrectly auto-rotated display on a mobile device held in his handwhile lying down on his side (e.g., on a bed);

FIG. 3B depicts an improved scenario in which the user views a correctlyauto-rotated display on a mobile device held in his hand while lyingdown on his side (e.g., on a bed);

FIG. 3C depicts a similar scenario as FIG. 3B, with the user lying downon his other side;

FIG. 4 is a high-level block diagram showing a system for detecting andutilizing the orientation of one or more objects in an image accordingto various embodiments of the disclosure;

FIG. 5 is a more detailed block diagram showing exemplary componentswithin an object orientation detector (such as the one shown in FIG. 4),in accordance with certain embodiments of the disclosure;

FIG. 6 illustrates an example of orientation-specific feature detectionperformed by an accelerated computer vision (CV) computation unit (suchas the one shown in FIG. 5), in accordance with certain embodiments ofthe disclosure;

FIG. 7 is a table showing a wide range of hypothesized feature anglesachieved via various combinations of (1) image rotation and (2) CV unitinput orientation, according to various embodiments;

FIG. 8 shows different states of an orientation detection controller,with each state representing a different combination of (1) imagerotation and (2) CV unit input orientation, according to variousembodiments;

FIG. 9 illustrates a hysteresis function built into the triggeroperation of an orientation tracking technique, in accordance withembodiments of the disclosure;

FIG. 10 is a flow chart showing illustrative steps in a process forperforming image processing and utilization according to at least oneembodiment of the disclosure; and

FIG. 11 illustrates an example computer system 1100 that can be used toimplement features of the disclosure.

DETAILED DESCRIPTION

Several illustrative embodiments will now be described with respect tothe accompanying drawings, which form a part hereof. While particularembodiments, in which one or more aspects of the disclosure may beimplemented, are described below, other embodiments may be used andvarious modifications may be made without departing from the scope ofthe disclosure or the spirit of the appended claims.

Illustrative Use Cases

FIG. 1A depicts a scenario in which a mobile device is laid flat on topof a table and viewed by a user. Here, the user is looking at thedisplay of the mobile device. The display is presenting an outdoor scenein “landscape” display rotation.

FIG. 1B is a different view of the scenario depicted in FIG. 1A, showingthe orientation of the display of the mobile device and the position ofthe user. As shown in the figure, the user is able to view the outdoorscene presented in “landscape” display rotation.

FIG. 2A depicts a subsequent scenario in which the mobile device remainsin the same position on the table, but the user has moved to view themobile device from a different user position, i.e., from a differentside of the table. In response to the new user position, the mobiledevice automatically adjusts the rotation of the display to show theoutdoor scene in a “portrait-like” display rotation, according toembodiments of the present disclosure. Not only is the display rotationaltered to be “portrait-like,” the display rotation matches the newposition of the user. That is, what the user sees is not an up-side-down“portrait” view, but a right-side-up “portrait” view of the outdoorscene. According to embodiments of the disclosure, the mobile devicecaptures an image of the user with a front-facing camera (i.e., camerafacing the user). The mobile device performs image processing todetermine the orientation of the face of user within the captured image.The determined orientation of the user's face is then used toautomatically adjust the rotation of the display, so that the properdisplay rotation is presented to the user. In this manner, the rotationof the display may “follow” the orientation of the user's face. Thus, inresponse to the user moving to the new user position (i.e., at adifferent side of the table), as depicted in FIG. 2A, the display isautomatically rotated to match the user's new viewing position.

FIG. 2B is a different view of the scenario depicted in FIG. 2A, showingthe orientation of the display of the mobile device, in response to thechanged position of the user. As shown in the figure, the display isautomatically rotated to present a right-side-up “portrait” view of theoutdoor scene.

FIG. 3A depicts a potentially frustrating scenario in which a user viewsan incorrectly auto-rotated display on a mobile device held in his handwhile lying down on his side (e.g., on a bed). Here, the mobile deviceutilizes a conventional technique for automatic display rotation, i.e.,one based on the direction of gravity as sensed by an accelerometer. Inthe scenario depicted in FIG. 3A, reliance on the direction of gravityfor automatic display rotation results in the incorrect rotation of thedisplay from the perspective of the user. In this case, the mobiledevice, using just the gravity vector, only senses that the mobiledevice has moved from an upright position to a prone position andconcludes that the display should be rotated (e.g., from “portrait” to“landscape” rotation) in response. However, the mobile device does notsense that the user has also moved from an upright to a prone position,and therefore no rotation of the display would actually be needed. Thus,the user is forced to read the text at an offset angle of 90 degrees.

FIG. 3B depicts an improved scenario in which the user views a correctlyauto-rotated display on a mobile device held in his hand while lyingdown on his side (e.g., on a bed). According to various embodiments ofthe disclosure, the mobile device uses an image of the user capturedfrom a front-facing camera and determines the orientation of the user'sface. Based on the determined orientation of the user's face within thecaptured image, the mobile device automatically adjusts the rotation ofthe display. For example, this results in the text being presented onthe display in the correct rotation for viewing by the user. The user isable to read the text without any appreciable offset angle.

FIG. 3C depicts a similar scenario as FIG. 3B, with the user lying downon his other side. Here again, the rotation of the display may “follow”the orientation of the user's face. Thus, even though the user has nowchanged his position to lie on the other side of his body, the textpresented on the display is still in the correct rotation for viewing bythe user. In yet another scenario (not shown), the user might be lyingon his back while holding up the mobile device directly above his face,with the display of the device pointed in a downward direction towardthe user's face. Here, if a conventional system based on direction ofgravity is used to automatically control the rotation of the display,the display may be incorrectly and unintentionally rotated even with theslightest movement of the user's hand holding the mobile device.According to various embodiments of the present disclosure, the detectedorientation of the user's face may be used, instead of the direction ofgravity, to determine the appropriate rotation of the display and thusavoid such incorrect and unintentional display rotation.

Overall System

FIG. 4 is a high-level block diagram showing a system 400 for detectingand utilizing the orientation of one or more objects in an imageaccording to various embodiments of the disclosure. As shown, the system400 comprises an image sensor 402, an object orientation detector 404,and an object orientation receiver 406. According to certainembodiments, system 400 resides on a mobile device, such as a handheldsmart phone device. Image sensor 402 captures an image of a scene. Theindividual sensor elements, e.g., pixels, of image sensor 402 may bealigned in a rectangular grid in some embodiments but not aligned in arectangular grid in other embodiments. The image is made available toobject orientation detector 404. Object orientation detector 404 mayhave various components for efficiently determining the orientation ofone or more objects within the image. One example of such an object isthe face of a user of the mobile device. Thus, object orientationdetector 404 may determine the rotational orientation (e.g., in degrees)of the face of the user within the image. Here, object orientationdetector 404 may be trained to detect an object in different ranges oforientations. In some embodiments, the range of orientations covers anentire rotation of 360 degrees. In other embodiments, the object is onlydetected in a narrow range of orientations, e.g., plus or minus 30degrees. The orientation of the one or more objects in the image is thenprovided to object orientation receiver 406. Object orientation receiver406 represents a plethora of possible components that may utilize thedetermined orientation of the one or more objects within the image, tocontrol one or more operations of the mobile device.

For example, the orientation of the user's face within the image, asdetermined by object orientation detector 404, may be used by objectorientation receiver 406 to provide automatic display rotation for themobile device. Here, the rotation of the display presented to the usermay “follow” the orientation of the user's face. For instance, if theuser tilts his head while viewing the display of the mobile device,while keeping the mobile device stationary, the display of the mobiledevice may rotate and follow the tilt of the user's face. Similarly, ifthe mobile device is laid flat on top of a table (e.g., as shown inFIGS. 1A and 2A), or if the mobile device is held by a user who is lyingon his side (e.g., as shown in FIGS. 3B and 3C), the determinedorientation of the user's face may be utilized in performing automaticdisplay rotation of the mobile device.

In certain embodiments, automatic display rotation may be limited toonly a few possible display rotation outcomes, such as rotations of 0degrees, 90 degrees, 180 degrees, and 270 degrees. Such limited displayrotation outcomes may be adopted, for example, for a display having arectangular shape display. In other embodiments, automatic displayrotation may result in a greater number of possible display rotationoutcomes. For example, different display rotations separated by finerincrements, e.g., 2 degrees, may be possible. Such a wider range ofdisplay rotation outcomes may be adopted, for example, for a displayhaving a circular or other non-rectangular shape. Furthermore, thedisplay rotation may be used in other ways. For example, the display mayprovide different information to the user depending on the rotation. Asanother example, a device such as a display filter, e.g., a polarizedfilter, may be changed depending on the display rotation. In onescenario, the direction of polarization of the display filter may bechanged to match the direction of polarization of glasses worn by theuser.

In a different example, the orientation of a Quick Response (QR) codewithin an image may be determined and utilized to implement a moreefficient automatic QR code reader. Current QR code readers generallymust be turned on manually to scan for a QR code. The alternative is toput the QR code reader in an always-on mode, which unnecessarilyconsumes computational resources and can quickly drain the batterycharge in a mobile device. The computational demands of a QR reader areexacerbated by the lack of information regarding the orientation of theQR code captured in an image. In other words, if the orientation of theQR code in the image were known, the QR code reader can be given a “headstart” and be able to read the QR code much more quickly andefficiently. According to various embodiments of the disclosure, objectorientation detector 404 may determine the orientation of a QR codewithin an image captured by image sensor 402. The determined orientationof the QR code may then be provided to object orientation receiver 406.In this case, object orientation receiver 406 may control a QR codereader (not shown) implemented within the mobile device. Objectorientation receiver 406 may automatically turn on the QR code readerand give the QR code reader a “head start” by providing the determinedorientation of the QR code. This allows the QR code reader to read theQR code automatically and in a more efficient manner.

Two-Stage Object Orientation Determination

FIG. 5 is a more detailed block diagram showing exemplary componentswithin an object orientation detector 404 (such as the one shown in FIG.4), in accordance with certain embodiments of the disclosure. As shown,object orientation detector 404 comprises an image buffer 502, arotation buffer 504, an accelerated computer vision (CV) computationunit 506, and an orientation detection controller 508. Here, a portionof an image captured by an image sensor is provided to and stored inimage buffer 502. The portion of image may comprise an entire image ascaptured by the image sensor (e.g., image sensor 402 from FIG. 4) or apart of such an image. The portion of image may then be rotatedaccording to a specified amount of rotation (e.g., degrees) and storedin rotation buffer 504. Here, the mechanism for performing the imagerotation is not shown, but its operation may be understood by one ofordinary skill in the art. The amount of image rotation may be specifiedusing control bits provided by orientation detection controller 508.Once the rotated image portion is obtained, it may be provided toaccelerated CV computation unit 506.

In one embodiment, the unrotated image portion may also be provideddirectly from image buffer 502 to accelerated CV computation unit 506.In such an embodiment, during operations in which no image rotation isneeded, rotation buffer 504 may be bypassed. In another embodiment, theunrotated image portion may be provided from rotation buffer 504(controlled to provide zero rotation) to accelerated CV computation unit506. In any event, the rotated and/or unrotated image portion isprovided to accelerated CV computation unit 506.

According to various embodiments, accelerated CV computation unit 506performs orientation-specific feature detection. In other words,accelerated CV computation unit 506 is capable of detecting a targetfeature at different specified orientations, as explained in more detailwith respect to FIG. 6, discussed below. Orientation detectioncontroller 508 may provide control bits to specify the orientation atwhich accelerated CV computation unit 506 is to perform featuredetection. Upon detection of the target feature, accelerated CVcomputation unit raises an object detection flag, which is provided toorientation detection controller 508.

Thus, orientation detection controller 508 controls the operation ofboth rotation buffer 504 and accelerated CV computation unit 506 in a“multi-stage” approach, e.g., a “two-stage” approach, according tovarious embodiments of the disclosure. In one stage, the rotation of theportion of image stored in rotation buffer 504 is controlled. In anotherstage, the orientation input to the accelerated CV computation unit 506is controlled. In the present embodiment, the control bits for bothstages are provided simultaneously. Thus, the term “two-stage” broadlyrefers to the two types of control provided and is not meant tonecessarily connote operation at two different times.

FIG. 6 illustrates an example of orientation-specific feature detectionperformed by an accelerated computer vision (CV) computation unit 506(such as the one shown in FIG. 5), in accordance with certainembodiments of the disclosure. Here, accelerated CV computation unit 506is capable of performing feature detection according to any one of fourpossible orientations, e.g., 0 degrees, 90 degrees, 180 degrees, and 270degrees. The orientation may be specified as an input to acceleratedcomputer vision (CV) computation unit 506, such as via control bitsprovided by orientation detection controller 508 as discussedpreviously.

Utilizing a component such as accelerated CV computation unit 506 toperform orientation-specific feature detection in this manner hassignificant advantages. To perform feature detection at, for example, a90 degree rotation, it is not necessary to first rotate the imageportion by 90 degrees prior to feature detection. Image rotation, suchas that provided by image rotation buffer 504, may be computationallyintensive and involve a large number of read and write cycles. Instead,the unrotated image portion may be fed to accelerated CV computationunit 506, and accelerated CV computation unit 506 may work directly onthe unrotated image portion to detect a target feature at a 90 degreerotation offset. Accelerated CV computation unit 506 is capable of doingso by internally performing an efficient coordinate mapping, toappropriately map pixels to different orientations selected from thelimited number of available orientations, for example, at 0 degrees, 90degrees, 180 degrees, and 270 degrees, as shown in FIG. 6. AcceleratedCV computation unit 506 thus performs orientation-specific featuredetection directly on a portion of image received from image buffer 502or rotation buffer 504.

FIG. 7 is a table showing a wide range of hypothesized feature anglesachieved via various combinations of (1) image rotation and (2) CV unitinput orientation, according to various embodiments. In this example,orientation detection controller 508 may achieve a range of hypothesizedfeature angles, i.e., from 0 to 360 degrees in 15 degree increments, bycontrolling both the (1) image rotation and (2) CV unit inputorientation. The possible image rotations shown in this example are: 0degrees, 15 degrees, 30 degrees, 45 degrees, 60 degrees, and 75 degrees.The possible CV unit input orientations in this example are: 0 degrees,90 degrees, 180 degrees, and 270 degrees. Here, “the CV unit inputorientation” refers to the orientation of the detector, i.e., theorientation in which the detector operates to detect the target feature.The “CV unit input orientation” in this example does not refer to achange in the orientation of the image portion provided as input to theCV unit. The combination of both image rotation and CV unit inputorientation allows orientation detection controller 508 to cycle throughall angles from 0 to 360 degrees in 15 degree increments. At each suchangle, feature detection may be performed using accelerated CVcomputation unit 506 to see if the target feature is detected. Uponreceipt of an indication that the target feature is detected, such asvia the object detection flag shown in FIG. 5, orientation detectioncontroller 508 may declare that the current feature angle, e.g., 135degrees, is the detected orientation of the object in the image.

FIG. 8 shows different states of orientation detection controller 508,with each state representing a different combination of (1) imagerotation and (2) CV unit input orientation, according to variousembodiments. For example, orientation detection controller 508 maysystematically cycle through the states depicted in FIG. 8, until theorientation of the object within the image is detected. As shown, theprocess may start in a state that corresponds to image rotation at 0degrees. While keeping the image rotation at 0 degrees, the CV unitinput orientation may be cycled through different values, such as 0degrees, 90 degrees, 180 degrees, and 270 degrees. Next, the process maymove to a state that corresponds to image rotation at 15 degrees. Whilekeeping the image rotation at 15 degrees, the CV unit input orientationmay be cycled through different values, such as 0 degrees, 90 degrees,180 degrees, and 270 degrees. A similar pattern of cycling throughavailable CV unit input orientations may be applied through differentimage rotations such as 30 degrees, 45 degrees, 60 degrees, and finally70 degrees, until the target feature is recognized, at which point theorientation of the object (e.g., human face) is determined.

Note that the particular order in which the various states are visitedmay be altered. FIG. 8 shows just one example, in which the processholds the image rotation at each angle while cycling through differentCV unit input orientations (e.g., 0 degrees, 90 degrees, 180 degrees,and 270 degrees). Alternatively, the process may hold the CV unit inputorientation at each value while cycling through different imagerotations (e.g., 0 degrees, 15 degrees, 30 degrees, 45 degrees, 60degrees, and 75 degrees).

Detection Mode v. Tracking Mode

According to various embodiments, the orientation of the at least oneobject within an image is determined based on multiple modes, themultiple modes comprising (a) a detection mode for detecting an initialorientation of the at least one object within an initial image and (b) atracking mode for tracking the orientation of the at least one objectwithin a subsequent image, using the detected initial orientation. Here,detection and tracking may involve related but somewhat differentoperations. In particular, detection of the orientation of an object inan image, as discussed in the above sections, is generally done withoutany prior knowledge of the possible orientation of the object. That is,the orientation of the object is equally likely to be at one angle asany other angle. Thus, detection of the object's orientation within theimage may be performed by systematically cycling through all possibleangles (or angle increments) until the target feature (e.g., human face)is detected.

By contrast, tracking of the orientation of an object in an image isgenerally performed with some knowledge of the possible orientation ofthe object. In particular, because tracking generally follows detection,a “last known orientation” may be available and taken into account inperforming orientation tracking. Just as an example, consider a sequenceof images comprising image 0, image 1, image 2, image 3, etc. capturedby image sensor 402. By processing image 0, object orientation detectionmay be performed. For example, an orientation of a face of a user may bedetermined by using image 0 and utilizing operating image buffer 502,rotation buffer 504, accelerated CV computation unit 506, andorientation detection controller 508 in a manner such as that discussedin the above sections.

Once the orientation of the face of the user in image 0 is detected, theorientation of the face in image 1 may simply be tracked, withoutperforming a full set of operations associated with orientationdetection. For example, the tracking technique may take advantage ofprior knowledge of the “last known orientation” of the user's facedetermined using image 0. Various techniques for tracing may beemployed. One exemplary tracking technique involves starting at the“last known orientation” and using it as a seed angle, then performingfeature detection at hypothesized angles that progressively extend inpositive and negative relative offsets from the seed angle. Forinstance, if the orientation of the user's face was determined to be atangle of 90 degrees based on orientation detection performed on image 0,tracking can be employed to attempt to detect a human face first at +15degrees offset from 90 degrees (i.e., 105 degrees), followed byattempted detection at −15 degrees offset from 90 degrees (i.e., 75degrees), followed by attempted detection at +30 degrees offset from 90degrees (i.e., 120 degrees), followed by attempted detection at −30degrees offset from 90 degrees (i.e., 60 degrees), and so on, until ahuman face is detected in image 1. Such use of a seed angle provides ashortcut to avoid systematically cycling through all possible featureangles, such as that associated with full orientation detection. Thus,tracking that takes advantage of existing knowledge of the “last knownorientation” can be significantly more efficient, when compared toorientation detection performed without any knowledge of the possibleorientation of the object within the image.

FIG. 9 illustrates a hysteresis function built into the triggeroperation of an orientation tracking technique, in accordance withembodiments of the disclosure. In one embodiment, tracking may betriggered upon any deviation from the current angle. For example, in aparticular image, if the user's face is not detected at the “last knownorientation” from the previous image, the tracking mode may immediatelybe triggered and begin to search through hypothesized angles in anattempt to detect the user's face. However, in certain scenarios, thefailure to immediately detect the orientation of the user's face at the“last known orientation” might have occurred in error, e.g., as resultof noise, etc. Thus, according to certain embodiments, a hysteresisfunction is adopted to prevent a first false positive from causing anunnecessary search through hypothesized angles.

According to further embodiments, non-image sensors may also be used toaid in the tracking of the orientation of an object in a sequence ofimages. One example of such a non-image sensor is an accelerometer onboard the mobile device. The accelerometer reading may indicate anangular rotation of the mobile device. Readings from the accelerometermay be used in a variety of ways. One use of the accelerometer readingis to provide an additional or alternative source of information fortriggering the tracking mode of the system. Another use of theaccelerometer reading is to provide a seed angle for the tracking modeonce it has been triggered.

Referring back to FIG. 9, the figure shows an accelerometer readingbeing used to (1) trigger tracking according to a hysteresis functionand (2) provide a seed angle for performing tracking. Here, if theaccelerometer reading indicates that the mobile device has rotated by 30degrees or more, then the tracking mode is triggered. Upon entering thetracking mode, the reading from the accelerometer (shown as +S or −S)may be used as a seed angle from which to start tracking operations.

Furthermore, other information may be used to aid in the tracking of theorientation of an object in a sequence of images, according to certainembodiments of the disclosure. For example, statistics can be takenrelating to each image, the object, and/or the areas near the object, toestimate the possible orientation of the object. Non-image sensorreadings can also be used in a similar manner, as discussed previously.Such alternatively or additionally obtained estimates of the orientationof the object may be used to trigger, seed, or otherwise aid in thetracking of the orientation of the object within one or more images.Statistics such as those discussed above can be generated from an imagedetector such as object orientation detector 404, a different imagedetector, or a non-image sensor.

FIG. 10 is a flow chart showing illustrative steps in a process 1000 forperforming image processing and utilization according to at least oneembodiment of the disclosure.

FIG. 11 illustrates an example computer system 1100 that can be used toimplement features of the disclosure. Computer system 1100 is showncomprising hardware elements that can be electrically coupled via a bus1102 (or may otherwise be in communication, as appropriate). Thehardware elements may include one or more processors 1104, includingwithout limitation one or more general-purpose processors and/or one ormore special-purpose processors (such as digital signal processingchips, graphics processing units 1122, and/or the like); one or moreinput devices 1108, which can include without limitation one or morecameras, sensors, a mouse, a keyboard, a microphone configured to detectultrasound or other sounds, and/or the like; and one or more outputdevices 1110, which can include without limitation a display unit suchas the device used in implementations of the invention, a printer and/orthe like. Additional cameras 1120 may be employed for detection ofuser's extremities and gestures. In some implementations, input devices1108 may include one or more sensors such as infrared, depth, and/orultrasound sensors. The graphics processing unit 1122 may be used tocarry out the method for real-time wiping and replacement of objectsdescribed above.

In some implementations of the implementations of the invention, variousinput devices 1108 and output devices 1110 may be embedded intointerfaces such as display devices, tables, floors, walls, and windowscreens. Furthermore, input devices 1108 and output devices 1110 coupledto the processors may form multi-dimensional tracking systems.

The computer system 1100 may further include (and/or be in communicationwith) one or more non-transitory storage devices 1106, which cancomprise, without limitation, local and/or network accessible storage,and/or can include, without limitation, a disk drive, a drive array, anoptical storage device, a solid-state storage device such as a randomaccess memory (“RAM”) and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable and/or the like. Such storage devices maybe configured to implement any appropriate data storage, includingwithout limitation, various file systems, database structures, and/orthe like.

The computer system 1100 might also include a communications subsystem1112, which can include without limitation a modem, a network card(wireless or wired), an infrared communication device, a wirelesscommunication device and/or chipset (such as a Bluetooth device, an802.11 device, a WiFi device, a WiMax device, cellular communicationfacilities, etc.), and/or the like. The communications subsystem 1112may permit data to be exchanged with a network, other computer systems,and/or any other devices described herein. In many implementations, thecomputer system 1100 will further comprise a non-transitory workingmemory 1118, which can include a RAM or ROM device, as described above.

The computer system 1100 also can comprise software elements, shown asbeing currently located within the working memory 1118, including anoperating system 1114, device drivers, executable libraries, and/orother code, such as one or more application programs 1116, which maycomprise computer programs provided by various implementations, and/ormay be designed to implement methods, and/or configure systems, providedby other implementations, as described herein. Merely by way of example,one or more procedures described with respect to the method(s) discussedabove might be implemented as code and/or instructions executable by acomputer (and/or a processor within a computer); in an aspect, then,such code and/or instructions can be used to configure and/or adapt ageneral purpose computer (or other device) to perform one or moreoperations in accordance with the described methods.

A set of these instructions and/or code might be stored on acomputer-readable storage medium, such as the storage device(s) 1106described above. In some cases, the storage medium might be incorporatedwithin a computer system, such as computer system 1100. In otherimplementations, the storage medium might be separate from a computersystem (e.g., a removable medium, such as a compact disc), and/orprovided in an installation package, such that the storage medium can beused to program, configure and/or adapt a general purpose computer withthe instructions/code stored thereon. These instructions might take theform of executable code, which may be executable by the computer system1100 and/or might take the form of source and/or installable code,which, upon compilation and/or installation on the computer system 1100(e.g., using any of a variety of generally available compilers,installation programs, compression/decompression utilities, etc.) thentakes the form of executable code.

Substantial variations may be made in accordance with specificrequirements. For example, customized hardware might also be used,and/or particular elements might be implemented in hardware, software(including portable software, such as applets, etc.), or both. Further,connection to other computing devices such as network input/outputdevices may be employed. In some implementations, one or more elementsof the computer system 1100 may be omitted or may be implementedseparate from the illustrated system. For example, the processor 1104and/or other elements may be implemented separate from the input device1108. In one implementation, the processor may be configured to receiveimages from one or more cameras that are separately implemented. In someimplementations, elements in addition to those illustrated in FIG. 4 maybe included in the computer system 1100.

Some implementations may employ a computer system (such as the computersystem 1100) to perform methods in accordance with the disclosure. Forexample, some or all of the procedures of the described methods may beperformed by the computer system 1100 in response to processor 1104executing one or more sequences of one or more instructions (which mightbe incorporated into the operating system 1114 and/or other code, suchas an application program 1116) contained in the working memory 1118.Such instructions may be read into the working memory 1118 from anothercomputer-readable medium, such as one or more of the storage device(s)1106. Merely by way of example, execution of the sequences ofinstructions contained in the working memory 1118 might cause theprocessor(s) 1104 to perform one or more procedures of the methodsdescribed herein.

The terms “machine-readable medium” and “computer-readable medium,” asused herein, refer to any medium that participates in providing datathat causes a machine to operate in a specific fashion. In someimplementations implemented using the computer system 1100, variouscomputer-readable media might be involved in providing instructions/codeto processor(s) 1104 for execution and/or might be used to store and/orcarry such instructions/code (e.g., as signals). In manyimplementations, a computer-readable medium may be a physical and/ortangible storage medium. Such a medium may take many forms, includingbut not limited to, non-volatile media, volatile media, and transmissionmedia. Non-volatile media include, for example, optical and/or magneticdisks, such as the storage device(s) 1106. Volatile media include,without limitation, dynamic memory, such as the working memory 1118.Transmission media include, without limitation, coaxial cables, copperwire, and fiber optics, including the wires that comprise the bus 1102,as well as the various components of the communications subsystem 1112(and/or the media by which the communications subsystem 1112 providescommunication with other devices). Hence, transmission media can alsotake the form of waves (including without limitation radio, acousticand/or light waves, such as those generated during radio-wave andinfrared data communications).

Common forms of physical and/or tangible computer-readable mediainclude, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip orcartridge, a carrier wave as described hereinafter, or any other mediumfrom which a computer can read instructions and/or code.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 1104for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by the computer system 1100. These signals,which might be in the form of electromagnetic signals, acoustic signals,optical signals and/or the like, are all examples of carrier waves onwhich instructions can be encoded, in accordance with variousimplementations of the invention.

The communications subsystem 1112 (and/or components thereof) generallywill receive the signals, and the bus 1102 then might carry the signals(and/or the data, instructions, etc. carried by the signals) to theworking memory 1118, from which the processor(s) 1104 retrieves andexecutes the instructions. The instructions received by the workingmemory 1118 may optionally be stored on a non-transitory storage deviceeither before or after execution by the processor(s) 1104.

It is understood that the specific order or hierarchy of steps in theprocesses disclosed is an illustration of exemplary approaches. Basedupon design preferences, it is understood that the specific order orhierarchy of steps in the processes may be rearranged. Further, somesteps may be combined or omitted. The accompanying method claims presentelements of the various steps in a sample order, and are not meant to belimited to the specific order or hierarchy presented.

The previous description is provided to enable any person skilled in theart to practice the various aspects described herein. Variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. Moreover, nothing disclosed herein is intended to bededicated to the public.

While some examples of methods and systems herein are described in termsof software executing on various machines, the methods and systems mayalso be implemented as specifically-configured hardware, such asfield-programmable gate array (FPGA) specifically to execute the variousmethods. For example, examples can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or in acombination thereof. In one example, a device may include a processor orprocessors. The processor comprises a computer-readable medium, such asa random access memory (RAM) coupled to the processor. The processorexecutes computer-executable program instructions stored in memory, suchas executing one or more computer programs. Such processors may comprisea microprocessor, a digital signal processor (DSP), anapplication-specific integrated circuit (ASIC), field programmable gatearrays (FPGAs), and state machines. Such processors may further compriseprogrammable electronic devices such as PLCs, programmable interruptcontrollers (PICs), programmable logic devices (PLDs), programmableread-only memories (PROMs), electronically programmable read-onlymemories (EPROMs or EEPROMs), or other similar devices.

Such processors may comprise, or may be in communication with, media,for example computer-readable storage media, that may store instructionsthat, when executed by the processor, can cause the processor to performthe steps described herein as carried out, or assisted, by a processor.Examples of computer-readable media may include, but are not limited to,an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the describedembodiments. However, it will be apparent to one skilled in the art thatthe specific details are not required in order to practice the describedembodiments. Thus, the foregoing descriptions of specific embodimentsare presented for purposes of illustration and description. They are notintended to be exhaustive or to limit the described embodiments to theprecise forms disclosed. It will be apparent to one of ordinary skill inthe art that many modifications and variations are possible in view ofthe above teachings.

The foregoing description of some examples has been presented only forthe purpose of illustration and description and is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Numerous modifications and adaptations thereof will be apparent to thoseskilled in the art without departing from the spirit and scope of thedisclosure.

Reference herein to an example or implementation means that a particularfeature, structure, operation, or other characteristic described inconnection with the example may be included in at least oneimplementation of the disclosure. The disclosure is not restricted tothe particular examples or implementations described as such. Theappearance of the phrases “in one example,” “in an example,” “in oneimplementation,” or “in an implementation,” or variations of the same invarious places in the specification does not necessarily refer to thesame example or implementation. Any particular feature, structure,operation, or other characteristic described in this specification inrelation to one example or implementation may be combined with otherfeatures, structures, operations, or other characteristics described inrespect of any other example or implementation.

Use herein of the word “or” is intended to cover inclusive and exclusiveOR conditions. In other words, A or B or C includes any or all of thefollowing alternative combinations as appropriate for a particularusage: A alone; B alone; C alone; A and B only; A and C only; B and Conly; and A and B and C

What is claimed is:
 1. A method for image processing and utilization,comprising: obtaining, using a mobile device, an image containing a faceof a user of the mobile device; determining, using the mobile device, anorientation of the face of the user within the image, wherein theorientation of the face of the user within the image is determined usingmultiple stages, the multiple stages comprising: (a) a rotation stagefor controlling a rotation applied to a portion of the image, togenerate a portion of rotated image; and (b) an orientation stage forcontrolling an orientation applied to orientation-specific featuredetection performed on the portion of rotated image; and utilizing thedetermined orientation of the face of the user as a control input tomodify a display rotation of the mobile device.
 2. The method of claim1, wherein the portion of rotated image is stored in a rotation buffer.3. The method of claim 1, wherein the orientation-specific featuredetection is performed using a computer vision (CV) computation unit. 4.The method of claim 1, wherein the orientation of the face of the userwithin the image is determined based on multiple modes, the multiplemodes comprising: (a) a detection mode for detecting an initialorientation of the face of the user within an initial image; and (b) atracking mode for tracking the orientation of the face of the userwithin a subsequent image, using the detected initial orientation. 5.The method of claim 4, wherein in the detection mode, the initialorientation of the face of the user within the image is detected byperforming feature detection at a first plurality of hypothesizedangles; and wherein in the tracking mode, the orientation of the face ofthe user within the subsequent image is tracked by performing featuredetection at a second plurality of hypothesized angles, the secondplurality being fewer than the first plurality.
 6. The method of claim4, wherein in the tracking mode, the orientation of the face of the userwithin the subsequent image is tracked upon detection of a triggercondition associated with on a non-image sensor.
 7. The method of claim6, wherein the non-image sensor comprises an accelerometer.
 8. Anapparatus for image processing and utilization, comprising: an imagesensor configured to obtain an image containing at least a face of auser of a mobile device; an object orientation detector configured todetermine an orientation of the face of the user within the image,wherein the object orientation detector is configured to determine theorientation of the face of the user within the image using multiplestages, the multiple stages comprising: (a) a rotation stage configuredto control a rotation applied to a portion of the image, to generate aportion of rotated image; and (b) an orientation stage configured tocontrol an orientation applied to orientation-specific feature detectionperformed on the portion of rotated image; and an object orientationreceiver configured to receive and organize the determined orientationof the face of the user as a control input to modify a display rotationof the mobile device.
 9. The apparatus of claim 8, further comprising: arotation buffer configured to store the portion of rotated image. 10.The apparatus of claim 8, further comprising: a computer vision (CV)computation unit configured to perform the orientation-specific featuredetection.
 11. The apparatus of claim 8, wherein the object orientationdetector is configured to determine the orientation of the face of theuser on multiple modes, the multiple modes comprising: (a) a detectionmode for detecting an initial orientation of the face of the user withinan initial image; and (b) a tracking mode for tracking the orientationof the face of the user within a subsequent image, using the detectedinitial orientation.
 12. The apparatus of claim 11, wherein in thedetection mode, the object orientation detector is configured to detectthe initial orientation of the face of the user within the image byperforming feature detection at a first plurality of hypothesizedangles; and wherein in the tracking mode, the object orientationdetector is configured to track the orientation of the face of the userwithin the subsequent image by performing feature detection at a secondplurality of hypothesized angles, the second plurality being fewer thanthe first plurality.
 13. The apparatus of claim 11, wherein in thetracking mode, the object orientation detector is configured to trackthe orientation of the face of the user within the subsequent imagebased upon detection of a trigger condition associated with on anon-image sensor.
 14. The apparatus of claim 13, wherein the non-imagesensor comprises an accelerometer.
 15. A non-transitorycomputer-readable medium having instructions embedded thereon for imageprocessing and utilization, the instructions, when executed by one ormore processing units, cause the one or more processing units to:obtaining, using a mobile device, an image containing a face of a userof the mobile device; determining, using the mobile device, anorientation of the face of the user within the image, wherein theorientation of the face of the user within the image is determined usingmultiple stages, the multiple stages comprising: (a) a rotation stagefor controlling a rotation applied to a portion of the image, togenerate a portion of rotated image; and (b) an orientation stage forcontrolling an orientation applied to orientation-specific featuredetection performed on the portion of rotated image; and utilizing thedetermined orientation of the face of the user as a control input tomodify a display rotation of the mobile device.
 16. The non-transitorycomputer-readable medium of claim 15, wherein the portion of rotatedimage is stored in a rotation buffer.
 17. The non-transitorycomputer-readable medium of claim 15, wherein the orientation-specificfeature detection is performed using a computer vision (CV) computationunit.
 18. The non-transitory computer-readable medium of claim 15,wherein the orientation of the face of the user within the image isdetermined based on multiple modes, the multiple modes comprising: (a) adetection mode for detecting an initial orientation of the face of theuser within an initial image; and (b) a tracking mode for tracking theorientation of the face of the user within a subsequent image, using thedetected initial orientation.
 19. The non-transitory computer-readablemedium of claim 18, wherein in the detection mode, the initialorientation of the face of the user within the image is detected byperforming feature detection at a first plurality of hypothesizedangles; and wherein in the tracking mode, the orientation of the face ofthe user within the subsequent image is tracked by performing featuredetection at a second plurality of hypothesized angles, the secondplurality being fewer than the first plurality.
 20. The non-transitorycomputer-readable medium of claim 18, wherein in the tracking mode, theorientation of the face of the user within the subsequent image istracked upon detection of a trigger condition associated with on anon-image sensor.
 21. The non-transitory computer-readable medium ofclaim 20, wherein the non-image sensor comprises an accelerometer.
 22. Asystem for image processing and utilization, comprising: means forobtaining, using a mobile device, an image containing a face of a userof the mobile device; means for determining, using the mobile device, anorientation of the face of the user within the image, wherein theorientation of the face of the user within the image is determined usingmultiple stages, the multiple stages comprising: (a) a rotation stagefor controlling a rotation applied to a portion of the image, togenerate a portion of rotated image; and (b) an orientation stage forcontrolling an orientation applied to orientation-specific featuredetection performed on the portion of rotated image; and means forutilizing the determined orientation of the face of the user as acontrol input to modify a display rotation of the mobile device.
 23. Thesystem of claim 22, wherein the portion of rotated image is stored in arotation buffer.
 24. The system of claim 22, wherein theorientation-specific feature detection is performed using a computervision (CV) computation unit.
 25. The system of claim 22, wherein theorientation of the face of the user within the image is determined basedon multiple modes, the multiple modes comprising: (a) a detection modefor detecting an initial orientation of the face of the user within aninitial image; and (b) a tracking mode for tracking the orientation ofthe face of the user within a subsequent image, using the detectedinitial orientation.
 26. The system of claim 25, wherein in thedetection mode, the initial orientation of the face of the user withinthe image is detected by performing feature detection at a firstplurality of hypothesized angles; and wherein in the tracking mode, theorientation of the face of the user within the subsequent image istracked by performing feature detection at a second plurality ofhypothesized angles, the second plurality being fewer than the firstplurality.
 27. The system of claim 25, wherein in the tracking mode, theorientation of the face of the user within the subsequent image istracked upon detection of a trigger condition associated with on anon-image sensor.
 28. The system of claim 27, wherein the non-imagesensor comprises an accelerometer.